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Technical Field 

This invention relates to image estimation methods to improve representation of 
geometric entities within images when noise and outliers contaminate images. 

Background of the Invention 

Structure fitting is an important procedure for the measurement of geometrical entities 
within a digital image because it allows for a non-discrete representation of the image 
geometrical entity. Using the structure fit representation of the geometrical entity enables 
accurate measurement. Measurement accuracy achieved with this method usually is 
better than the pixel pitch contained in the original image. Such accuracy depends on 
accurate structure fitting to the features in an image. The traditional method of structure 
fitting is to extract a feature mask image from the original image and then apply the 
structure fitting technique using the mask portion of the image. This method is sensitive 
to the noise or distortion in the original image because they corrupt the mask. To 
overcome this problem, a weight image with weights corresponding to feature signal 
intensity is created from the original image and the weight image is used for structure 
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fitting. This method improves the robustness of the fitting, however, in the presence of 
strong noise or outlier, this method is not reliable. Under the prior art method, the noise 
and outliers degrade measurement accuracy and repeatability more than is necessary. 

Prior Art 

In the prior art approach, high quality of the feature weight image produces a good 
feature fitting result. Yet prior art suffers the limitations imposed by imperfect images. 
This invention improves on the prior art by dealing with a broad range of input image 
imperfections that include noise and outliers. 

Objects and Advantages 

It is an object of this invention to improve the robustness of image measurement. Non- 
contact video or image inspection accuracy and repeatability presently suffers from 
imperfect images. 

It is an object of this invention to use learning images to improve measurement 
robustness. Learning improves fitting to geometric entities by utilizing learned 
knowledge of expected feature information and variations. 

It is an object of this invention to iteratively improve image measurement. Iteration can 
progressively reduce the effect of image impairment and thereby increase accuracy and 
repeatability of measurement. 

Summary of the Invention 

In an industrial application, the measurement of geometrical shape and geometrical entity 
is important for non-contact video or image inspection. When the image is noise free, the 
feature mask image has good quality and therefore the result of structure fitting is 
accurate. However, if the image is contaminated by noise and/or is distorted or 
aberrated, the quality of the feature mask image used for structure fitting may be poor. 
The feature mask image could include outlier points well beyond the desired geometric 
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entities. A low quality feature mask image degrades the structure fitting result. An 
alternative arrangement for measurement creates a weight image for fitting. The objective 
is to create a weight image such that the pixel value of the weight image indicates the 
degree of certainty or feature signal strength to guide the structure fitting. The fitting 
method using the weight image is therefore less sensitive to image noise and distortion. 

Two embodiments of the invention are described. The first embodiment updates a 
weight image using the information generated from learning. The second embodiment 
uses iterative adjustment. For the iterative method, the weight image is adjusted where 
the fitting error is big. 
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Brief Description of the Drawings 



The preferred embodiments and other aspects of the invention will become apparent from 
the following detailed description of the invention when read in conjunction with the 
accompanying drawings which are provided for the purpose of describing embodiments 
of the invention and not for limiting same, in which: 

Figure 1 shows the basic procedure for measurement; 
Figure 2 shows the measurement procedure using a weight image; 
Figure 3 shows a structure guided measurement procedure; 
Figure 4 shows a method for accumulating a weight image; 
Figure 5 shows creation of a mixed weight image; 

Figure 6 shows creation of a mixed weight image from the deviation images and weight 
images; 

Figure 7 shows a method to derive the intra-deviation image from the weight image; 

Figure 8 shows the procedure for iterative modification of the weight image for fitting; 

Figure 9A shows an example perfect line image; 

Figure 9B shows outliers to add to the example image; 

Figure 9C shows background noise to add to the example image; 

Figure 9D shows the example line image with outliers and image blurring; 

Figure 9E shows the degraded image created by adding random noise and background 

noise to the image in Figure 9D; 

Figure 9F shows the mask image that is extracted from the image; 
Figure 9G shows the extracted weight image used for fitting; 

Figure 9H shows the first iteration fitting result by overlaying the fitted line with the 
distorted image; 

Figure 91 shows the fitting result after 3 iterations; 

Figure 9 J shows the fitting result after 6 iterations; 

Figure 9K shows the fitting result after 10 iterations; 

Figure 10 shows the processing flow for creating a feature mask image; 
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Figure 1 1 shows the processing flow of the post threshold morphology operation used in 
Figure 10. 
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Detailed Description of the Invention 

I. Objective 

In industrial applications, measurements of geometrical shape and geometrical entity are 
important for non-contact video or image inspection. These measurements are done 
using images acquired from an imaging system such as a camera with appropriate 
illumination and optics. As shown in Figure 1, the measurement procedure includes three 
steps. An input image is received 100. Image processing for enhancement may already 
have been performed on the received image. From this input image 100, a feature mask 
image is generated 102. The feature mask image contains potential geometric entities of 
interest. Using the mask image, structure fitting methods are performed 104. 
Measurement 106 is performed using the structure fitted result. Such structure-guided 
image processing and measurement methods are disclosed in a co-pending U.S. Patent 
Application Ser. No. 09/739084 entitled, "Structure Guided Image Measurement 
Method", by Shih-Jong J. Lee et. al., filed December 14, 2000 which is incorporated 
herein in its entirety. 

One method for generating mask images is described in a co-pending U.S. Patent 
Application Ser. No. 09/738846 entitled, "Structure-guided Image Processing and Image 
Feature Enhancement" by Shih-Jong J. Lee, filed December 15, 2000 which is 
incorporated herein in its entirety. Mask images are generated using an image 
thresholding step applied to the feature enhanced image to generate binary level masks of 
the geometric entities regions of interest. In one embodiment of the invention, boundary 
masks are generated using a general edge detection method derived from the difference 
of binary dilation and erosion. A connected component labeling step (ref: U.S. Patent 
Application Ser. No. 09/702629, "Run-Length Based Image Processing Programmed in a 
Computer", by Shih-Jong J. Lee, filed October 31, 2000) is applied to the boundary 
masks to assign a unique label for each connected component of the mask image. 

When the image is noise free, the mask image should have good quality and therefore the 
result of the fitting should be accurate. However, if an image is contaminated by noise 
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or is distorted or aberrated, the quality of the mask image may be poor. A low quality 
feature mask image degrades the structure fitting result. An alternative embodiment for 
measurement uses a feature mask image and a weight image for structure fitting as shown 
in Figure 2. An image is received 200 and a weight image is created 210 along with a 
feature mask for the image 202 to indicate the area of interest within the image. Using 
the mask image and the weight image, a structure is fitted 204 and measurements are 
determined 206 from the fitted geometric entities to produce measurement results 208. 
The objective is to create a weight image 210 whose pixel value indicates the degree of 
certainty or feature signal strength for the fitted structure. Methods for creating such a 
weight image are described in co-pending U.S. Patent Application Ser. No. 09/738846 
entitled, "Structure-guided Image Processing and Image Feature Enhancement" by Shih- 
Jong J. Lee, filed December 15, 2000. Because the weight image indicates the degree of 
certainty or feature signal strength, a fitting method using the weight image is less 
sensitive to image noise and distortion. However, in the presence of stronger noise, the 
quality of the feature mask image and the weight image could be degraded significantly. 
The feature mask image could include outlier points well beyond the desired geometric 
entities. An example of a line corrupted by outliers and noise is provided in Detailed 
Description of the Invention - Section V Example. In the example, the outliers and noise 
cause a poor quality feature mask image and weight image so the initial fitting result is 
incorrect. This invention provides methods to overcome these difficulties by updating 
the weight image. Two embodiments of the invention are described. The first 
embodiment performs adjustment using the information from learning results, and the 
second embodiment uses iterative adjustment. 

II. Geometrical entity fitting using the weight image 

Figure 3 shows the procedure for structure fitting using a feature mask image and a 
weight image. The feature mask image 300 and the weight image 302 are used in 
combination 304 to create best fitting structure parameter values 306. Let the weight 

image be i w . From this image, the best fit to the structure f(x, y\ Q) = 0 can be 
performed by selecting Q * that minimizes the cost function 
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cost = £ £ U] [y,] * [f( Xi , y, ; Q)f 

m ieC 

^ m 

Where: Q is the parameter vector and Iwixiliyil is the pixel values of the weight image 
at pixel (jc, , y.) . There are m geometric entities and C m is the feature mask region of the 

m th geometric entity. The best fit is to find the vector Q * that yields the minimum value 
of cost. The result of this method is highly dependent on the weight image. Good 
result can be achieved if the corruption of the image is small. However, when image 
distortion or noise is significant, the resulting fit can be degraded. When the feature 
mask image contains outliers of the geometrical entity, the result can be much worse 

since the cost function includes a square of fix^y^Q) . 

This invention provides a robust method that reduces the effects of noise, distortion, and 
outliers by adjusting the weight image. A first method is to adjust the weight image 
uses learning information, and a second method adjusts the weight image iteratively. 

III. Weight image adjustment using learning images 

Learning images are acquired from the expected applications. The learning method 
accumulates the information contained within the feature mask regions of the learning 
images. One embodiment of the invention accumulates the learning images as shown in 
Figure 4. A learning image 400 is received and weighted by (1- a) 402. The weighted 
learning image is added to the previous weight image result 406 that has been further 
weighted by a factor a 404 to produce a new weight image 408. The mask image could 
be dilated prior to performing the accumulation to avoid misalignment. In one 
embodiment of the invention, a weighted mean image and a weighted mean of square 
image are accumulated. The weighted mean image and the weighted mean of square 
image can be updated by the following rule. 



Confidential 



Page 9 



5/31/2001 



I W, inter (* + 1) = « I WJnter (n) + (1 ~ /„ (/I + 1) 

ATiv,tor (n + 1) = a ^ W(toer (n) + (1 - a) / u 2 (w + 1) 

where the value a (0 < a < 1 ) determines the contribution ratio of the accumulated and 
the new image, /„ (n + 1) is the n+1 m learning image, I WMter {n) is the weighted mean 

image result for accumulation up to the n th learning image. l Wtinter (n + 1) is the weighted 
mean image result for accumulation up to the n+l th learning image. Kwjnter ( n ) is the 
weighted mean of the square image result for accumulation up to the n th learning image. 
Kw, inter (w + 1) is the weighted mean of square image result for accumulation up to the 
n+l th learning image. 

To achieve exponential weighted moving average, a can be set to a constant. This 
emphasizes newer learning images. In this case, the order of learning is important. It is 
used when the noise process is changing with respect to image sequence and more recent 
images provide better knowledge about the next image. 

Alternatively, a simple equal weight average can be achieved by setting: 

n 

a ~ 

n + 1 

In this case, a increases with each learning iteration. This is used when there is no value 
to order in the learning process. 

A deviation image I D (n)of the weight image for up to the n th learning image can be 
derived by the following rule: 



iDMter ( n ) — *\J Kw, inter ( n ) I W, inter ( n ) 
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The deviation image is a measure of the local (i.e. pixel) variability of the image across 
the n learning images. 

In one embodiment of the invention, the mask image is generated when the fitting 
method is applied to a first learning image and the initial condition is set as follows: 

Iw.»aerQ) = Iu(\) 
Kw Anter 

In the application phase, a weight image derived only from the input image (intra-weight 
image) can be mixed with the weight image derived from the learning result (inter-weight 
image) as shown in Figure 5. In Figure 5 an inter- weight image 500 is mixed with an 
intra-weight image 502 using a mixing method 504 to produce a mixed weight image 
result 506. Several mixing methods 504 can be used. One mixing method 504 is the 
minimum operation: 

I W, intra [x][y]). 

A high value of I w means high certainty, so this method picks the local alternative (i.e. 
pixel level) that weights information with the maximum represented uncertainty 
(minimum represented certainty). 

Those skilled in the art should recognize that other mixing operations 504 could be used 
including a maximum operation or a simple average operation or other operations. 

If an intra-deviation image is available, the mixing method can use the information from 
the deviation image as shown in Figure 6. In Figure 6, an inter-weight image 606 is 
mixed with an intra-weight image 602, an inter-deviation image 604 and an intra- 
deviation image 600 using a mixing method 608 to produce a mixed weight image 610. 

In one embodiment of the invention, the mixing method 608 can be implemented by the 
following rule: 
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r ir I Winter 

[x][y] if / Dtfater [x][y] < I DMtra [x][y] 
Uw,m/raW[> ; ] otherwise 



This method picks the local alternative (i.e. pixel alternative) for greatest certainty of the 
weight image. 

An alternative embodiment of the invention, the method 608 can be implemented by the 
following rule: 

_ r i r i I WMer[ X ][y] * I D Antra [x][y) + i W Antra 

IwWiyi : 

I D Antra Wlyl+lDAnterWIy] 

□ 

:? This method gives softer mixing rather than discontinuous hard mixing selection in the 

VB 

H 4 previous method. Therefore, it could yield more robust estimates of the weight image. A 

\f% smaller pixel value of the deviation image indicates a higher certainty of the 

;« corresponding pixel in the weight image. To apply this mixing method, the intra- 

s deviation image 600 of the weight image / 0)in , rfl W[y] needs to be generated. The 

t as* 

\f\ generation of the intra-deviation image of the weight image uses neighboring information 
V*. as shown in Figure 7. The weight image pixel 700 is combined with the average value of 
□ neighboring pixels 702 by subtraction. The absolute difference values are used 704 and 
the average value of neighboring pixels replaces the individual pixel 706 to produce an 
intra-deviation image result 708. This method reduces high spatial frequency noise. In a 
preferred embodiment, 8 nearest neighbors are used for averaging. Those skilled in the 
art should recognize that the number of neighbors used depends upon the noise power 
spectrum. 

Other methods could be used to match the nature and the distribution of the noise power 
spectrum. 

IV. Iterative adjustment of weight image 
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This iterative fitting method adjusts the weight image as shown in Figure 8. A weight 
image 800 is received and adjusted 802 pixel by pixel based upon a scale factor 808 for 
each pixel determined from the prior fitting result. Using the adjusted weight image, a 
new fitting result 806 is generated by parameter estimation 804. In the first fitting 
iteration, the adjustment operation is not performed. It is performed in all subsequent 
fitting iterations. 

This method provides better fitting results by using the previous fitting result to adjust the 
weight image. If the previous fitting error measured by the cost function defined in 
Detailed Description of the Invention - Section II (the fitting error is measured 
individually, pixel by pixel) is large, the pixel value of the adjusted weight image will be 
smaller. This effectively minimizes the effect of the pixels having the large fitting errors. 
In a preferred embodiment, the weight image is adjusted by a multiplication factor that is 
a non-increasing function of the fitting error. Mathematically, 

Iw,n+x [x][y] = I W M [y] * r(\f (x, y\ Q)\j) 

where Wty] * s the adjusted weight image at the n th iteration. g„ IS * e parameter 

vector which is the result of the n th iteration. r(z;T) is a non-increasing function. 
Examples of r(z;T) include 

r(z;r) = exp(-^), 

r(z;r) = exp(— ^-),or 
T 

r(z\T) = - 1 



»+&) 



The above non-increasing function has a parameter T whose value determines the speed 
of convergence during the iteration. T can be a constant for all iterations or it can vary 
during the iterations. If T is large, then the iteration could converge to the global 
minimum of the cost function, but the convergence speed is slow. Conversely, if T is 
small, the convergence speed is fast, but the end result is not guaranteed to converge to 
the global minimum. To achieve a good balance, the T value can vary with the number 
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of iterations. In one embodiment of the invention, the simulated annealing method 
(Stuart Geman et. al., "Stochastic Relaxation, Gibbs Distributions, and the Bayesian 
Restoration of Images", IEEE Transactions on Pattern Analysis and Machine 
Intelligence, Vol. PAMI-6, No. 6, November 1984, pp. 721-741) can be used. To 
achieve good performance, T should be non-increasing with the number of iterations 
One good choice is 



Those skilled in the art should recognize that there are many other functional relations 
between T and n. For example: 



V. Stopping Criteria 

In one embodiment of the invention, if the maximum allowable error of the parameter Q 
at each iteration is defined as £ , then the stopping criteria is 



The operation |a-b| can be Li (norm 1), L 2 (norm 2), and L~ (norm °°) between a and b. 
An alternative embodiment of the invention uses a maximum allowable iteration N as the 
stopping criteria. That is 

If the number of iterations = N, then stop. 

Another alternative embodiment of the invention uses both the maximum allowable error 
and the maximum allowable iteration for the stopping criteria. The weight adjustment 
iteration will be stopped if either one of the criteria is met. 



r ^~Xog(n + l) 



where 



n is the number of iterations. 




Q + i~<2 <€ 
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VI. Example 

Figures 9 A-K show an example for iteratively adjusting the weight image. To construct 
the degraded image (Figure 9E) } an example line image (Figure 9A) is degraded by 
outliers (Figure 9B) and background noise (Figure 9C) and then blurred (Figure 9D) with 
random noise added to become Figure 9E. Blurring and noise are physical effects 
common to imaging systems. The resulting example image is composed of the line 
segment with two outliers and significant image spatial noise. The feature mask image 
(Figure 9F) is extracted from the degraded image (Figure 9E) using the process shown in 
Figure 10 and Figure 11. Creation of feature mask images is described in co-pending U.S. 
Patent Application Ser. No. 09/738846 entitled, "Structure-guided Image Processing and 
Image Feature Enhancement" by Shih-Jong J. Lee, filed December 15, 2000 which is 
incorporated in its entirety herein. Figures 10 and 1 1 further describe the process used in 
this particular example, to create the mask image. An edge enhancement process consists 
of subtraction of the result of a cascade of fitting operations 1002 from the input image 
1000 that creates a weight image 1004. In this embodiment, the cascade open-close 
operation is applied by increasing the structure elements from size 3 to size 23 with an 
increment of 2 in each iteration. That is: open(3,3) — » close(3,3) — > open(5,5) — > 
close(5,5) -» ... open(23,23) -> close(23,23). An opening 1006 and closing 1008 
operation removes residual image noise from the weight image 1004 before thresholding 
1010 is applied. After thresholding, the process shown in Figure 11, 1014, is used to 
remove artifacts that corrupt the thresholded result. Figure 1 1 further describes the post 
threshold morphology operation 1014. In this particular example embodiment, the 
thresholded image 1 100 is opened 1 102, closed 1 104, and the result is filtered twice using 
elongated filters 1106, 1114 and then combined by union (OR operation) 1108 and 
further closed 1110 to produce the mask image 1016. The resulting mask image 1016 is 
still somewhat corrupted by noise and outliers. Using the mask image, the weight image 
is extracted from the original image. The extracted image used for fitting is shown in 
Figure 9G. In this example embodiment, fitting is done using the single line fitting 
method described in co-pending U.S. Patent Application Ser. No. 09/739084 entitled, 
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"Structure Guided Image Measurement Method", by Shih-Jong J. Lee et. ah, filed 
December 14, 2000 which is incorporated in its entirety herein. There is one geometric 
entity in the feature mask image and the function to minimize is ax + by + c = 0 where a, 
b and c are the parameters of the line with a 2 + b 2 = 1. The r(z;T) selected for this example 
is a monotonically decreasing function as follows: 

r(z;r) = exp(-^/) 

where 

z = \ax + by + c| 
and T = 2 

Figure 9H shows the first fitting result. The outliers and the image noise corrupt this 
result. Figure 91 shows the fitting result after three iterations. Figure 9 J shows the fitting 
result after six iterations. Figure 9K shows the fitting result after ten iterations. After ten 
iterations the result has a minimal effect caused by the outliers and image noise. The 
robustness of the iterative method with an image contaminated by noise, blurring, and 
outliers is demonstrated. 

The invention has been described herein in considerable detail in order to comply with 
the Patent Statutes and to provide those skilled in the art with the information needed to 
apply the novel principles and to construct and use such specialized components as are 
required. However, it is to be understood that the inventions can be carried out by 
specifically different equipment and devices, and that various modifications, both as to 
the equipment details and operating procedures, can be accomplished without departing 
from the scope of the invention itself. 
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